Life After the EM Algorithm: The Variational Approximation for Bayesian Inference

Thomas Bayes (1701-1761), shown in the upper left, first discovered " Bayes' theorem " in a paper that was published in 1764 three years after his death, as the name suggests. However, Bayes in his theorem used uniform priors[1]. Pierre-Simon Laplace (1749-1827), shown in the lower right, apparently unaware of Bayes' work, discovered the same theorem in more general form in a memoir he wrote at the age of 25 and showed its wide applicability [2]. Regarding these issues S. M. Stiegler writes in [3] " The influence of this memoir was immense. It was from here that " Bayesian " ideas first spread through the mathematical world, as Bayes's own article was ignored until 1780 and played no important role in scientific debate until the twentieth century. It was also this article of Laplace's that introduced the mathematical techniques for the asymptotic analysis of posterior distributions that are still employed today. And it was here that the earliest example of optimum estimation can be found, the derivation and characterization of an estimator that minimized a particular measure of posterior expected loss. After more than two centuries, we mathematicians, statisticians cannot only recognize our roots in this masterpiece of our science, we can still learn from it. " 2 Abstract Maximum Likelihood (ML) estimation is one of the most popular methodologies used in modern statistical signal processing. The Expectation Maximization (EM) algorithm is an iterative algorithm for ML estimation that has a number of advantages and has become a standard methodology for solving statistical signal processing problems. However, the EM has certain requirements that seriously limit its applicability to complex problems. Recently, a new methodology termed " variational Bayesian inference " has emerged, which relaxes some of the limiting requirements of the EM algorithm and is gaining rapidly popularity. Furthermore, one can show that the EM algorithm can be viewed as a special case of this methodology. In this paper we first present a tutorial introduction of Bayesian variational inference aimed at the signal processing community. Then, we use linear regression and Gaussian mixture modeling as examples to demonstrate the additional capabilities that Bayesian variational inference offers as compared to the EM algorithm.